Backup/Restore: add support for external compressors and decompressors by rsajwani · Pull Request #10558 · vitessio/vitess

rsajwani · 2022-06-21T14:33:56Z

Description

Replacement PR for #7978
Signed-off-by: Renan Rangel renan@slack-corp.com

This change will allow us to use an external decompressor when available as opposed to the internal pgzip currently used. We ran some tests using pigz -d -c and we get a nice speed bump on decompression, around 30% or more faster decompressing when restoring from a backup in S3. While decompressing is still single-threaded in pigz, it uses separate threads for reading, writing and checksum, and it results in faster performance.

I wanted to get an initial feedback on this change, as we would like to also to plug our own compressor to use other encryption algorithms (like lz4 or zstd) with an external binary as this is a more flexible setup for users, but also work on the change to add built-in support for one of these compression algorithms.

Related Issue(s)

#7802

Checklist

"Backport me!" label has been added if this change should be backported
Tests were added or are not required
Documentation was added or is not required - separate website PR is planned

Deployment Notes

github-actions · 2022-06-27T17:01:44Z

Signed-off-by: Renan Rangel <renan@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

Signed-off-by: Renan Rangel <rrangel@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

go/test/endtoend/backup/xtrabackup/xtrabackup_test.go

go/test/endtoend/backup/mysqlctld/backup_mysqlctld_test.go

go/test/endtoend/backup/xtrabackup/xtrabackup_test.go

go/vt/mysqlctl/compression.go

go/vt/wrangler/testlib/backup_test.go

test/config.json

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

go/vt/mysqlctl/compression.go

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

doc/releasenotes/15_0_0_summary.md

go/flags/endtoend/vttablet.txt

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

deepthi

Looks like there was one place missed when flags were renamed in the original PR.

go/vt/mysqlctl/xtrabackupengine.go

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

…itessio#950) * Backup/Restore: add support for external compressors and decompressors (vitessio#10558) * change to support an external decompressor Signed-off-by: Renan Rangel <renan@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * add external compressor support + builtin additional compressors Signed-off-by: Renan Rangel <renan@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * wrap external compressor/decompressor Signed-off-by: Renan Rangel <renan@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * go mod tidy + comments Signed-off-by: Renan Rangel <renan@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * add copyright notices Signed-off-by: Renan Rangel <renan@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * add support for builtin engine Signed-off-by: Renan Rangel <rrangel@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Adding test case for buckup compression Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fixing unit test and run mod tidy Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Removing unwanted unit tests Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Increase timeout of backup tests Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * fixing linter errors Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Change test logic to accomodate running selective tests Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * removing lint warning Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * fixing test failure Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Removing un-necessary test Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fixing code review feeback Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Change builtinEngine to consider 'auto' decompressor Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * fixing Upgrade/Downgrade test Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fix type & add summary under release notes Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fixing typos in summary Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fixing flag name typos Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> Co-authored-by: Renan Rangel <rrangel@slack-corp.com> Co-authored-by: Renan Rangel <renan@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * cherry-pick some how didn't took the right code for test Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Remove built-in decompression flag (vitessio#10670) * Remove built-in decompression flag Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fix test failures Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fix Helpoutput test Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fixing unit test Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Adding summary Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * code cleaning and better summary Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Change builtinCompressor to more generic compression engine name Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fixing / Adding new test case Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fix summary & static code analysis Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Adding fake backup impl in test Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Adding time sleep in between test Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fixing summary and adding comments Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Feedback on summary Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Code review feedback Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fixing comment Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fixing default value in summary Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fixing test cases Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * More summary fixes Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fixing TestHelpOutput Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> Co-authored-by: Renan Rangel <rrangel@slack-corp.com> Co-authored-by: Renan Rangel <renan@slack-corp.com>

vitessio#10558) * change to support an external decompressor Signed-off-by: Renan Rangel <renan@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * add external compressor support + builtin additional compressors Signed-off-by: Renan Rangel <renan@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * wrap external compressor/decompressor Signed-off-by: Renan Rangel <renan@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * go mod tidy + comments Signed-off-by: Renan Rangel <renan@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * add copyright notices Signed-off-by: Renan Rangel <renan@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * add support for builtin engine Signed-off-by: Renan Rangel <rrangel@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Adding test case for buckup compression Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fixing unit test and run mod tidy Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Removing unwanted unit tests Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Increase timeout of backup tests Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * fixing linter errors Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Change test logic to accomodate running selective tests Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * removing lint warning Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * fixing test failure Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Removing un-necessary test Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fixing code review feeback Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Change builtinEngine to consider 'auto' decompressor Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * fixing Upgrade/Downgrade test Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fix type & add summary under release notes Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fixing typos in summary Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fixing flag name typos Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> Co-authored-by: Renan Rangel <rrangel@slack-corp.com> Co-authored-by: Renan Rangel <renan@slack-corp.com>

…test + compressors (#119) * Delete all legacy sharding related code (vitessio#10278) * Delete all legacy sharding related code Signed-off-by: Matt Lord <mattalord@gmail.com> * Move used until functions from initialsharding to cluster Signed-off-by: Matt Lord <mattalord@gmail.com> * Remove vtctl commands Signed-off-by: Matt Lord <mattalord@gmail.com> * Kill vtworker and SetKeyspaceServedFrom cmd Signed-off-by: Matt Lord <mattalord@gmail.com> * WaitForDrain related stragglers Signed-off-by: Matt Lord <mattalord@gmail.com> * Legacy local straggler workflow Signed-off-by: Matt Lord <mattalord@gmail.com> * Get rid of SetKeyspaceShardingInfo & wait for drain stragglers Signed-off-by: Matt Lord <mattalord@gmail.com> * Remove vtworker stragglers Signed-off-by: Matt Lord <mattalord@gmail.com> * Update throttlerservice protobuf Signed-off-by: Matt Lord <mattalord@gmail.com> * Rename test 24, add hashicorp vault test to it (now mysql_server_vault) Signed-off-by: Matt Lord <mattalord@gmail.com> * Remove last mentiones of legacy sharding in vtctl Signed-off-by: Matt Lord <mattalord@gmail.com> * remove binlog_use_v3_resharding_mode Signed-off-by: Matt Lord <mattalord@gmail.com> * Address review comments Signed-off-by: Matt Lord <mattalord@gmail.com> * Address review comments Signed-off-by: Matt Lord <mattalord@gmail.com> * Correct vtgate help output Signed-off-by: Matt Lord <mattalord@gmail.com> * go fmt Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Remove v2 resharding fields (vitessio#10409) * cleanup: remove sharding_column_name and sharding_column_type Signed-off-by: Arvind Murty <arvind.murty@gmail.com> * cleanup: remove sharding_column_name and sharding_column_type Signed-off-by: Arvind Murty <arvind.murty@gmail.com> * cleanup: remove sharding_column_name and sharding_column_type Signed-off-by: Arvind Murty <arvind.murty@gmail.com> * cleanup: remove sharding_column_name and sharding_column_type Signed-off-by: Arvind Murty <arvind.murty@gmail.com> * generate vtadmin files Signed-off-by: Arvind Murty <arvind.murty@gmail.com> * cleanup: remove sharding_column_name and sharding_column_type from vtadmin Signed-off-by: Arvind Murty <arvind.murty@gmail.com> * Merge from main Signed-off-by: Arvind Murty <arvind.murty@gmail.com> * Fix bad merge conflict resolution Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Fix missing 'sharding' import Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Fix bad conflict resolution in go/test/endtoend/cellalias/cell_alias_test.go Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Backup/Restore: add support for external compressors and decompressors (vitessio#10558) * change to support an external decompressor Signed-off-by: Renan Rangel <renan@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * add external compressor support + builtin additional compressors Signed-off-by: Renan Rangel <renan@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * wrap external compressor/decompressor Signed-off-by: Renan Rangel <renan@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * go mod tidy + comments Signed-off-by: Renan Rangel <renan@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * add copyright notices Signed-off-by: Renan Rangel <renan@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * add support for builtin engine Signed-off-by: Renan Rangel <rrangel@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Adding test case for buckup compression Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fixing unit test and run mod tidy Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Removing unwanted unit tests Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Increase timeout of backup tests Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * fixing linter errors Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Change test logic to accomodate running selective tests Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * removing lint warning Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * fixing test failure Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Removing un-necessary test Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fixing code review feeback Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Change builtinEngine to consider 'auto' decompressor Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * fixing Upgrade/Downgrade test Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fix type & add summary under release notes Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fixing typos in summary Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> * Fixing flag name typos Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com> Co-authored-by: Renan Rangel <rrangel@slack-corp.com> Co-authored-by: Renan Rangel <renan@slack-corp.com> * Add MySQL 8 Support to Backup Tests (vitessio#10691) * Add support for MySQL 8.0 in backup tests Signed-off-by: Matt Lord <mattalord@gmail.com> * Add 8.0 workflow Signed-off-by: Matt Lord <mattalord@gmail.com> * whitespace Signed-off-by: Matt Lord <mattalord@gmail.com> * Use vtctldclient SetKeyspaceDurabilityPolicy to manage semi-sync This needed to be done after the shard was setup in order to satisfy the semantic assumptions related to semi-sync in the tests. Signed-off-by: Matt Lord <mattalord@gmail.com> * Remove extraneous changes Signed-off-by: Matt Lord <mattalord@gmail.com> * We need lz4 for TestXtrabackupStreamWithlz4Compression Signed-off-by: Matt Lord <mattalord@gmail.com> * Try using Percona Repo for MySQL 8 to align mysqld and xtrabackup versions Signed-off-by: Matt Lord <mattalord@gmail.com> * Specify stream type everywhere Signed-off-by: Matt Lord <mattalord@gmail.com> * Remove repeated server install Signed-off-by: Matt Lord <mattalord@gmail.com> * Moar... Signed-off-by: Matt Lord <mattalord@gmail.com> * Move vtctlbackup test to 8.0 Signed-off-by: Matt Lord <mattalord@gmail.com> * Rename vtbackup test and move to MySQL 8 Signed-off-by: Matt Lord <mattalord@gmail.com> * Split the xbstream tests so the workflow doesn't time out Otherwise it was going over the 10min limit and getting killed. Signed-off-by: Matt Lord <mattalord@gmail.com> * Use MySQL 8 compat method for setting passwords Signed-off-by: Matt Lord <mattalord@gmail.com> * Test increasing timeout at another level Signed-off-by: Matt Lord <mattalord@gmail.com> * Don't use the init passwords file with 8.0 mysqlctl doesn't start... This is likely due to the change in initialization behavior with MySQL 8.0 as it goes though two phases and you can't simply start up mysqld and pass it data, it has to initialize and restart first. Signed-off-by: Matt Lord <mattalord@gmail.com> * Fix incorrect password update statement for vt_repl user Signed-off-by: Rohit Nayak <rohit@planetscale.com> * Bump timeouts for 8.0 backup tests Signed-off-by: Matt Lord <mattalord@gmail.com> * Bump it more :( Signed-off-by: Matt Lord <mattalord@gmail.com> * Increase backup/restore timeout in backup_utils Signed-off-by: Matt Lord <mattalord@gmail.com> * Apply new 8.0 template everywhere Signed-off-by: Matt Lord <mattalord@gmail.com> * Fix bugs around how the compression flags were getting passed Signed-off-by: Matt Lord <mattalord@gmail.com> * Use 45m timeout for the workflow, 30m for the run. Signed-off-by: Matt Lord <mattalord@gmail.com> * These changes were no longer needed so limiting diff. Signed-off-by: Matt Lord <mattalord@gmail.com> * Explicitly skip new linter check Signed-off-by: Matt Lord <mattalord@gmail.com> Co-authored-by: Rohit Nayak <rohit@planetscale.com> * Fix test file merge issues Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Go 1.18.7 to fix error Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Fix lint Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Fix vttablet.txt Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * Remove sharding_column_name and sharding_column_type from vtctld2 (vitessio#10459) * Remove sharding_column_name and sharding_column_type Signed-off-by: notfelineit <notfelineit@gmail.com> * Run make web_build to compile production files Signed-off-by: notfelineit <notfelineit@gmail.com> * Remove vars from test Signed-off-by: notfelineit <notfelineit@gmail.com> * Remove sharding column type select box test from web test Signed-off-by: notfelineit <notfelineit@gmail.com> --------- Signed-off-by: Matt Lord <mattalord@gmail.com> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Signed-off-by: Arvind Murty <arvind.murty@gmail.com> Signed-off-by: notfelineit <notfelineit@gmail.com> Co-authored-by: Matt Lord <mattalord@gmail.com> Co-authored-by: Arvind Murty <arvind.murty@gmail.com> Co-authored-by: Rameez Sajwani <rameezwazirali@hotmail.com> Co-authored-by: Renan Rangel <rrangel@slack-corp.com> Co-authored-by: Renan Rangel <renan@slack-corp.com> Co-authored-by: Rohit Nayak <rohit@planetscale.com> Co-authored-by: Frances Thai <31225471+notfelineit@users.noreply.github.com>

…test + compressors (#119) (#124) * Delete all legacy sharding related code (vitessio#10278) * Delete all legacy sharding related code * Move used until functions from initialsharding to cluster * Remove vtctl commands * Kill vtworker and SetKeyspaceServedFrom cmd * WaitForDrain related stragglers * Legacy local straggler workflow * Get rid of SetKeyspaceShardingInfo & wait for drain stragglers * Remove vtworker stragglers * Update throttlerservice protobuf * Rename test 24, add hashicorp vault test to it (now mysql_server_vault) * Remove last mentiones of legacy sharding in vtctl * remove binlog_use_v3_resharding_mode * Address review comments * Address review comments * Correct vtgate help output * go fmt * Remove v2 resharding fields (vitessio#10409) * cleanup: remove sharding_column_name and sharding_column_type * cleanup: remove sharding_column_name and sharding_column_type * cleanup: remove sharding_column_name and sharding_column_type * cleanup: remove sharding_column_name and sharding_column_type * generate vtadmin files * cleanup: remove sharding_column_name and sharding_column_type from vtadmin * Merge from main * Fix bad merge conflict resolution * Fix missing 'sharding' import * Fix bad conflict resolution in go/test/endtoend/cellalias/cell_alias_test.go * Backup/Restore: add support for external compressors and decompressors (vitessio#10558) * change to support an external decompressor * add external compressor support + builtin additional compressors * wrap external compressor/decompressor * go mod tidy + comments * add copyright notices * add support for builtin engine * Adding test case for buckup compression * Fixing unit test and run mod tidy * Removing unwanted unit tests * Increase timeout of backup tests * fixing linter errors * Change test logic to accomodate running selective tests * removing lint warning * fixing test failure * Removing un-necessary test * Fixing code review feeback * Change builtinEngine to consider 'auto' decompressor * fixing Upgrade/Downgrade test * Fix type & add summary under release notes * Fixing typos in summary * Fixing flag name typos * Add MySQL 8 Support to Backup Tests (vitessio#10691) * Add support for MySQL 8.0 in backup tests * Add 8.0 workflow * whitespace * Use vtctldclient SetKeyspaceDurabilityPolicy to manage semi-sync This needed to be done after the shard was setup in order to satisfy the semantic assumptions related to semi-sync in the tests. * Remove extraneous changes * We need lz4 for TestXtrabackupStreamWithlz4Compression * Try using Percona Repo for MySQL 8 to align mysqld and xtrabackup versions * Specify stream type everywhere * Remove repeated server install * Moar... * Move vtctlbackup test to 8.0 * Rename vtbackup test and move to MySQL 8 * Split the xbstream tests so the workflow doesn't time out Otherwise it was going over the 10min limit and getting killed. * Use MySQL 8 compat method for setting passwords * Test increasing timeout at another level * Don't use the init passwords file with 8.0 mysqlctl doesn't start... This is likely due to the change in initialization behavior with MySQL 8.0 as it goes though two phases and you can't simply start up mysqld and pass it data, it has to initialize and restart first. * Fix incorrect password update statement for vt_repl user * Bump timeouts for 8.0 backup tests * Bump it more :( * Increase backup/restore timeout in backup_utils * Apply new 8.0 template everywhere * Fix bugs around how the compression flags were getting passed * Use 45m timeout for the workflow, 30m for the run. * These changes were no longer needed so limiting diff. * Explicitly skip new linter check * Fix test file merge issues * Go 1.18.7 to fix error * Fix lint * Fix vttablet.txt * Remove sharding_column_name and sharding_column_type from vtctld2 (vitessio#10459) * Remove sharding_column_name and sharding_column_type * Run make web_build to compile production files * Remove vars from test * Remove sharding column type select box test from web test --------- Signed-off-by: Matt Lord <mattalord@gmail.com> Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Signed-off-by: Arvind Murty <arvind.murty@gmail.com> Signed-off-by: notfelineit <notfelineit@gmail.com> Co-authored-by: Matt Lord <mattalord@gmail.com> Co-authored-by: Arvind Murty <arvind.murty@gmail.com> Co-authored-by: Rameez Sajwani <rameezwazirali@hotmail.com> Co-authored-by: Renan Rangel <rrangel@slack-corp.com> Co-authored-by: Renan Rangel <renan@slack-corp.com> Co-authored-by: Rohit Nayak <rohit@planetscale.com> Co-authored-by: Frances Thai <31225471+notfelineit@users.noreply.github.com>

rsajwani self-assigned this Jun 23, 2022

rsajwani added Component: Backup and Restore Type: Feature labels Jun 23, 2022

rsajwani marked this pull request as ready for review June 27, 2022 17:01

rsajwani requested review from deepthi, mattlord and rohit-nayak-ps as code owners June 27, 2022 17:01

rvrangel and others added 15 commits June 27, 2022 17:12

change to support an external decompressor

69db835

Signed-off-by: Renan Rangel <renan@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

add external compressor support + builtin additional compressors

0a00401

Signed-off-by: Renan Rangel <renan@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

wrap external compressor/decompressor

c71fd0a

Signed-off-by: Renan Rangel <renan@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

go mod tidy + comments

3c3180f

Signed-off-by: Renan Rangel <renan@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

add copyright notices

9f61d73

Signed-off-by: Renan Rangel <renan@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

add support for builtin engine

a210501

Signed-off-by: Renan Rangel <rrangel@slack-corp.com> Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

Adding test case for buckup compression

8c295f7

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

Fixing unit test and run mod tidy

a155689

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

Removing unwanted unit tests

e6a8456

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

Increase timeout of backup tests

38186ad

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

fixing linter errors

95afc73

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

Change test logic to accomodate running selective tests

2f72baa

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

removing lint warning

9f7812a

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

fixing test failure

46fde1d

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

Removing un-necessary test

a2d96fa

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

rsajwani force-pushed the ExtDecompressor branch from 64d2b06 to a2d96fa Compare June 28, 2022 00:12

deepthi reviewed Jun 28, 2022

View reviewed changes

go/test/endtoend/backup/xtrabackup/xtrabackup_test.go Outdated Show resolved Hide resolved

deepthi requested review from ajm188 and deepthi June 28, 2022 01:36

deepthi reviewed Jun 28, 2022

View reviewed changes

Fixing code review feeback

13f5e07

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

deepthi reviewed Jun 29, 2022

View reviewed changes

Fix type & add summary under release notes

1fc44e0

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

deepthi reviewed Jul 6, 2022

View reviewed changes

deepthi added release notes and removed release notes (needs details) This PR needs to be listed in the release notes in a dedicated section (deprecation notice, etc...) labels Jul 6, 2022

Merge remote-tracking branch 'upstream/main' into ExtDecompressor

606a33a

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

rsajwani force-pushed the ExtDecompressor branch from 01e1f50 to 606a33a Compare July 6, 2022 21:45

deepthi mentioned this pull request Jul 6, 2022

Update backup/restore docs vitessio/website#1089

Closed

rsajwani added 2 commits July 6, 2022 17:40

Fixing typos in summary

b6acb3e

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

Fixing flag name typos

2674adb

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

frouioui removed the release notes label Jul 7, 2022

Merge remote-tracking branch 'upstream/main' into ExtDecompressor

59fe698

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

rvrangel approved these changes Jul 7, 2022

View reviewed changes

rvrangel mentioned this pull request Jul 7, 2022

Add documentation for new backup compression options vitessio/website#1091

Merged

deepthi reviewed Jul 7, 2022

View reviewed changes

go/vt/mysqlctl/xtrabackupengine.go Outdated Show resolved Hide resolved

Merge remote-tracking branch 'upstream/main' into ExtDecompressor

8daed1e

Signed-off-by: Rameez Sajwani <rameezwazirali@hotmail.com>

deepthi merged commit 3256526 into vitessio:main Jul 7, 2022

deepthi deleted the ExtDecompressor branch July 7, 2022 19:16

deepthi mentioned this pull request Jul 12, 2022

Support for additional compressors/decompressors #7978

Closed

2 tasks

rvrangel mentioned this pull request Aug 17, 2022

revert default compression engine #11029

Merged

3 tasks

tanjinx mentioned this pull request Sep 29, 2022

Revert backup compressor from pargzip to pgzip slackhq/vitess#11

Merged

3 tasks

This was referenced Jan 5, 2023

Bug Report: vtbackup is missing compression/decompression flags #12047

Closed

backup: remove deprecated hook support #12066

Merged

timvaillancourt mentioned this pull request Aug 17, 2023

slack-vitess-r14.0.5: remove legacy sharding, add MySQL 8.x backup test + compressors slackhq/vitess#119

Merged

4 tasks

Conversation

rsajwani commented Jun 21, 2022 • edited by deepthi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

Checklist

Deployment Notes

Uh oh!

github-actions bot commented Jun 27, 2022 • edited by deepthi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review Checklist

General

Bug fixes

Non-trivial changes

New/Existing features

Backward compatibility

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

deepthi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rsajwani commented Jun 21, 2022 •

edited by deepthi

Loading

github-actions bot commented Jun 27, 2022 •

edited by deepthi

Loading